{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributed Tracing Template\n",
    "\n",
    " Illustrate the configuration for allowing distributed tracing using Jaeger.\n",
    " \n",
    "## Dependencies\n",
    "\n",
    " * [Helm](https://github.com/kubernetes/helm)\n",
    " * [Minikube](https://github.com/kubernetes/minikube)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test using Minikube\n",
    "\n",
    "**Due to a [minikube/s2i issue](https://github.com/SeldonIO/seldon-core/issues/253) you will need [s2i >= 1.1.13](https://github.com/openshift/source-to-image/releases/tag/v1.1.13)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!minikube start --memory 4096 --feature-gates=CustomResourceValidation=true --extra-config=apiserver.Authorization.Mode=RBAC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm init"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Jaeger\n",
    "\n",
    "We will use the Jaeger All-in-1 resource found at the [Jaeger Kubernetes repo](https://github.com/jaegertracing/jaeger-kubernetes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Jaeger UI\n",
    "\n",
    "```\n",
    "minikube service jaeger-query -n seldon\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-crd --name seldon-core-crd  --set usage_metrics.enabled=true --namespace seldon\n",
    "!helm install ../../../helm-charts/seldon-core --name seldon-core --namespace seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Jaeger ConfigMap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize tracing-configmap.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "configmap/tracing-config created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f tracing-configmap.yaml -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Example REST Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"labels\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"app\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"tracing-example\"\u001b[39;49;00m,\r\n",
      "\t\u001b[34;01m\"namespace\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\t\r\n",
      "    },\r\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"tracing-example\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_key\"\u001b[39;49;00m: \u001b[33m\"oauth-key\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_secret\"\u001b[39;49;00m: \u001b[33m\"oauth-secret\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "                \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\r\n",
      "                    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "                        \u001b[34;01m\"containers\"\u001b[39;49;00m: [\r\n",
      "                            {\r\n",
      "                                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"model1\"\u001b[39;49;00m,\t\t\t\t\r\n",
      "                                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/mock_classifier_rest:1.1\"\u001b[39;49;00m,\r\n",
      "\t\t\t\t\u001b[34;01m\"env\"\u001b[39;49;00m: [\r\n",
      "\t\t\t\t    {\r\n",
      "\t\t\t\t\t\u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"TRACING\"\u001b[39;49;00m,\r\n",
      "\t\t\t\t\t\u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "\t\t\t\t    },\r\n",
      "\t\t\t\t    {\r\n",
      "\t\t\t\t\t\u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_CONFIG_PATH\"\u001b[39;49;00m,\r\n",
      "\t\t\t\t\t\u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"/etc/tracing/config/tracing.yml\"\u001b[39;49;00m\r\n",
      "\t\t\t\t    }\r\n",
      "\t\t\t\t],\r\n",
      "\t\t\t\t\u001b[34;01m\"volumeMounts\"\u001b[39;49;00m: [\r\n",
      "\t\t\t\t    {\r\n",
      "\t\t\t\t\t\u001b[34;01m\"mountPath\"\u001b[39;49;00m: \u001b[33m\"/etc/tracing/config\"\u001b[39;49;00m,\r\n",
      "\t\t\t\t\t\u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"tracing-config\"\u001b[39;49;00m\r\n",
      "\t\t\t\t    }\r\n",
      "\t\t\t\t]\r\n",
      "                            }\r\n",
      "\t\t\t],\r\n",
      "\t\t\t\u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "\t\t\t\u001b[34;01m\"volumes\"\u001b[39;49;00m: [\r\n",
      "\t\t\t    {\r\n",
      "\t\t\t\t\u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"tracing-config\"\u001b[39;49;00m,\r\n",
      "\t\t\t\t\u001b[34;01m\"volumeSource\"\u001b[39;49;00m : {\r\n",
      "\t\t\t\t    \u001b[34;01m\"configMap\"\u001b[39;49;00m: {\r\n",
      "\t\t\t\t\t\u001b[34;01m\"localObjectReference\"\u001b[39;49;00m :\r\n",
      "\t\t\t\t\t{\r\n",
      "\t\t\t\t\t    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"tracing-config\"\u001b[39;49;00m\r\n",
      "\t\t\t\t\t},\r\n",
      "\t\t\t\t\t\u001b[34;01m\"items\"\u001b[39;49;00m: [\r\n",
      "\t\t\t\t\t    {\r\n",
      "\t\t\t\t\t\t\u001b[34;01m\"key\"\u001b[39;49;00m: \u001b[33m\"tracing.yml\"\u001b[39;49;00m,\r\n",
      "\t\t\t\t\t\t\u001b[34;01m\"path\"\u001b[39;49;00m:  \u001b[33m\"tracing.yml\"\u001b[39;49;00m\r\n",
      "\t\t\t\t\t    }\r\n",
      "\t\t\t\t\t]\r\n",
      "\t\t\t\t    }\r\n",
      "\t\t\t\t}\r\n",
      "\t\t\t    }\r\n",
      "\t\t\t]\r\n",
      "\t\t    }\r\n",
      "\t\t}],\r\n",
      "                \u001b[34;01m\"graph\"\u001b[39;49;00m: {\r\n",
      "\t\t    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"model1\"\u001b[39;49;00m,\r\n",
      "\t\t    \u001b[34;01m\"endpoint\"\u001b[39;49;00m: { \u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"REST\"\u001b[39;49;00m },\r\n",
      "\t\t    \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m,\r\n",
      "\t\t    \u001b[34;01m\"children\"\u001b[39;49;00m: [\r\n",
      "\t\t    ]\r\n",
      "\t\t},\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"tracing\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "\t\t\u001b[34;01m\"svcOrchSpec\"\u001b[39;49;00m : {\r\n",
      "\t\t    \u001b[34;01m\"env\"\u001b[39;49;00m: [\r\n",
      "\t\t\t{\r\n",
      "\t\t\t    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"TRACING\"\u001b[39;49;00m,\r\n",
      "\t\t\t    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "\t\t\t},\r\n",
      "\t\t\t{\r\n",
      "\t\t\t    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_AGENT_HOST\"\u001b[39;49;00m,\r\n",
      "\t\t\t    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"jaeger-agent\"\u001b[39;49;00m\r\n",
      "\t\t\t},\r\n",
      "\t\t\t{\r\n",
      "\t\t\t    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_AGENT_PORT\"\u001b[39;49;00m,\r\n",
      "\t\t\t    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"5775\"\u001b[39;49;00m\r\n",
      "\t\t\t},\r\n",
      "\t\t\t{\r\n",
      "\t\t\t    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_SAMPLER_TYPE\"\u001b[39;49;00m,\r\n",
      "\t\t\t    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"const\"\u001b[39;49;00m\r\n",
      "\t\t\t},\r\n",
      "\t\t\t{\r\n",
      "\t\t\t    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_SAMPLER_PARAM\"\u001b[39;49;00m,\r\n",
      "\t\t\t    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "\t\t\t}\r\n",
      "\t\t    ]\t\t\t\t\r\n",
      "\t\t}\r\n",
      "            }\r\n",
      "        ]\r\n",
      "    }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize deployment_rest.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/tracing-example created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create -f deployment_rest.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get sdep tracing-example -o jsonpath='{.status}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------\n",
      "SENDING NEW REQUEST:\n",
      "{'meta': {}, 'data': {'names': ['feat1', 'feat2', 'feat3'], 'ndarray': [[-1.7476463302482368, 1.0403015758607754, -0.5845863387322653]]}}\n",
      "Getting token from http://192.168.39.75:31888/oauth/token\n",
      "{\"access_token\":\"5dfe7536-83ab-4882-90c2-4fe357fe5964\",\"token_type\":\"bearer\",\"expires_in\":42525,\"scope\":\"read write\"}\n",
      "RECEIVED RESPONSE:\n",
      "{'meta': {'puid': '9imi6g2q72iujm0qqffhd9bpj9', 'tags': {}, 'routing': {}, 'requestPath': {'model1': 'seldonio/mock_classifier_rest:1.1'}, 'metrics': []}, 'data': {'names': ['proba'], 'ndarray': [[0.033983350227398276]]}}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!seldon-core-api-tester contract.json \\\n",
    "    `minikube ip` `kubectl get svc -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].spec.ports[0].nodePort}'` \\\n",
    "    --oauth-key oauth-key --oauth-secret oauth-secret -p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the Jaeger UI. You should be able to find traces like below:\n",
    "\n",
    "<img src=\"jaeger-ui-rest-example.png\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"tracing-example\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f deployment_rest.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Example GRPC Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f deployment_grpc.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get sdep tracing-example -o jsonpath='{.status}' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon-core-api-tester contract.json \\\n",
    "    `minikube ip` `kubectl get svc -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].spec.ports[1].nodePort}'` \\\n",
    "    --oauth-key oauth-key --oauth-secret oauth-secret -p --grpc \\\n",
    "    --oauth-port `kubectl get svc -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].spec.ports[0].nodePort}'`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the Jaeger UI. You should be able to find traces like below:\n",
    "\n",
    "<img src=\"jaeger-ui-grpc-example.png\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl delete -f deployment_grpc.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
